Evolutionary Training of Sparse Artificial Neural Networks: A Network Science Perspective

摘要

Through the success of deep learning, Artificial Neural Networks (ANNs) areamong the most used artificial intelligence methods nowadays. ANNs have led tomajor breakthroughs in various domains, such as particle physics, reinforcementlearning, speech recognition, computer vision, and so on. Taking inspirationfrom the network properties of biological neural networks (e.g. sparsity,scale-freeness), we argue that (contrary to general practice) Artificial NeuralNetworks (ANN), too, should not have fully-connected layers. We show how ANNsperform perfectly well with sparsely-connected layers. Following a Darwinianevolutionary approach, we propose a novel algorithm which evolves an initialrandom sparse topology (i.e. an Erd\H{o}s-R\'enyi random graph) of twoconsecutive layers of neurons into a scale-free topology, during the ANNtraining process. The resulting sparse layers can safely replace thecorresponding fully-connected layers. Our method allows to quadratically reducethe number of parameters in the fully conencted layers of ANNs, yieldingquadratically faster computational times in both phases (i.e. training andinference), at no decrease in accuracy. We demonstrate our claims on twopopular ANN types (restricted Boltzmann machine and multi-layer perceptron), ontwo types of tasks (supervised and unsupervised learning), and on 14 benchmarkdatasets. We anticipate that our approach will enable ANNs having billions ofneurons and evolved topologies to be capable of handling complex real-worldtasks that are intractable using state-of-the-art methods.